Methodological considerations for estimating policy effects in the context of co-occurring policies.

Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999 to 2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (> 82%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models.

Methodological considerations for estimating policy effects in the context of co-occurring policies

Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999 to 2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (> 82%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias;however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models.

Assessment of State and Federal Health Policies for Opioid Use Disorder Treatment During the COVID-19 Pandemic and Beyond.

IMPORTANCE: Federal and state governments implemented temporary strategies for providing access to opioid use disorder (OUD) treatment during the COVID-19 pandemic. Advocates hope many of these policies become permanent because of their potential to expand access to care. OBJECTIVE: To consider the multitude of ways access to and utilization of treatment for individuals with OUD might have been expanded by state and federal policy so researchers can do a better job evaluating the effectiveness of specific policy approaches, which will depend on the interaction with other state policies. EVIDENCE REVIEW: We summarize state-level policy data reported by government and nonprofit agencies that track health care regulations, specifically the Kaiser Family Foundation, Federation of State Medical Boards, American Association of Nurse Practitioners, American Academy of Physician Assistants, and the National Safety Council. Data were collected by these sources from September 2020 through January 2021. We examine heterogeneity in policy elements adopted across states during the COVID-19 pandemic in 4 key areas: telehealth, privacy, licensing, and medication for opioid use disorder. The analysis was conducted from March 2020 through January 2021. FINDINGS: This cross-sectional study found that federal and state governments have taken important steps to ensure OUD treatment availability during the COVID-19 pandemic, but few states are comprehensive in their approach. Although all states and Washington, DC have adopted at least 1 telehealth policy, only 17 states have adopted telehealth policies that improve access to OUD treatment for new patients. Furthermore, only 9 states relaxed privacy laws, which influence the ability to use particular technology for telehealth visits. Similarly, all states have adopted at least 1 policy related to health care professional licensing permissions, but only 35 expanded the scope of practice laws for both physician assistants and nurse practitioners. Forty-four states expanded access to initiation and delivery of medication for OUD treatment. Together, no state has implemented all of these policies to comprehensively expand access to OUD treatment during the COVID-19 pandemic. CONCLUSIONS AND RELEVANCE: With considerable policy changes potentially affecting access to treatment and treatment retention for patients with OUD during the pandemic, evaluations must account for the variation in state approaches in related policy areas because the interactions between policies may limit the potential effectiveness of any single policy approach.